Image processing device, image processing method, and integrated circuit for processing images

ABSTRACT

This image processing apparatus, for photographed images taken at a predetermined time interval and input sequentially, specifies an image area as the target of predetermined processing. The apparatus (i) has processing capability to generate, in accordance with a particular input photographed image, reduced images at K (K≧1) ratios within the predetermined time interval, (ii) selects, for each photographed image that is input, M (M≦K) or fewer ratios from among L (L&gt;K) different ratios in accordance with ratios indicated for a photographed image input prior to the photographed image, (iii) compares each of the reduced images generated at the selected M or fewer ratios with template images, and (iv) in accordance with the comparison results, specifies the image area.

TECHNICAL FIELD

The present invention relates to image processing technology, and inparticular to technology for specifying in a photographed image an imagearea matching predetermined criteria.

BACKGROUND ART

Established technology exists for performing predetermined processing tospecify an area in a photographed image (frame image) taken atpredetermined intervals (e.g. every 33 ms) by digital cameras or thelike, the area including a specified image (e.g. a facial image) smallerthan the size of the photographed image, and to display a frame on thespecified area, to focus on the specified area, etc.

One method for specifying an area including a specified image is todetect the specified image by comparing a template image (an image of apredetermined size, smaller than the photographed image) for detectingthe specified image with a plurality of reduced images based on thephotographed image, which are generated by changing the reduction ratioincrementally. In other words, a specified image is detected byperforming comparative processing for each reduced image while shiftingthe template image from the upper left pixel of the rectangular reducedimage to the lower right pixel.. Based on the results of thisprocessing, the area containing the specified image is identified.

The reduction ratios are preset so as to make it possible to detectspecified images of varying sizes included in a photographed image bycomparing the generated plurality of reduced images with the templateimage.

Such processing to generate reduced images, however, generally requiresa relatively long period of time, and as the number of reduced imagesgenerated from one photographed image grows larger, it may becomeimpossible to detect a specified image within the time interval betweenphotographs (in this example, 33 ms).

In particular, when a moving subject is being photographed, and forexample photographs are displayed at the same rate as the photographyframe rate, then if for a certain photographed image, a specified imageis detected after the time interval between photographs has elapsed, thespecified image area and the actual area in the subject may become outof alignment to a relatively large degree. For example, if predeterminedprocessing is performed to display a frame around the specified imagearea, the location of the frame may end up differing from the locationin the actual subject.

To solve this problem, there is a known method of improving theprobability of detecting a specified image within the time intervalbetween photographs by changing the order in which reduced images aregenerated in accordance with the photography mode of the digital camera(for example, Patent Document 1).

In the imaging apparatus in Patent Document 1, it is possible togenerate reduced images G1-G7 that are incrementally reduced from theoriginal image G0 (in this example, the size decreases sequentially fromG1 to G7). On the other hand, if the camera is in a mode forphotographing people or other such mode in which close-range photographyof a small number of people can be expected, then since the size of thefacial images in the photographed image tends to be larger in thesemodes, the reduced images can be generated in the order from G7 to G0and compared with the template image. Furthermore, if the camera is in amode for photographing scenery or other such mode in which long-rangephotography can be expected, then since the size of the facial images inthe photographed image tends to be smaller in these modes, the reducedimages can be generated in the order from G0 to G7 and compared with thetemplate image.

As a result, even if comparison of all of the reduced images with thetemplate is not complete within the time interval between photographsand processing must proceed to the next photographed image, it ispossible to improve the probability of detecting a specified imagewithin the time interval between photographs.

[Citation List] [Patent Literature] [Patent Literature 1]

Japanese Patent Application Publication No. 2007-135115

SUMMARY OF INVENTION Technical Problem

In recent years, however, users' needs for photography at a high framerate have grown. When responding to these needs, however, theabove-described problem becomes even more prevalent.

That is, when a specified image is detected after the time intervalbetween photographs has elapsed, the specified image area and the actualarea in the subject can more easily wind up out of alignment to arelatively large degree.

One way of solving this problem is through high-speed processing usinghigh specification hardware that can generate each reduced image in ashorter period of time. This approach, however, requires a relativelylarge change in design, making an approach that uses current hardwareresources preferable.

In light of the above-described problems, therefore, it is an object ofthe present invention to provide an image processing apparatus that candetect a specified image within the time interval between photographsvia a non-conventional method while using current hardware resources.

Solution to Problem

The above-described object is fulfilled by an image processing apparatusfor specifying, in photographed images taken at a predetermined timeinterval and input sequentially, an image area as a target ofpredetermined processing, the image processing apparatus comprising: areduced image generation unit that (i) has processing capability togenerate reduced images, in accordance with a particular inputphotographed image, at K (K≧1) ratios within the predetermined timeinterval and (ii) is operable to generate reduced images at ratiosindicated from among L (L>K) different ratios; a comparison unitoperable to compare each reduced image generated by the reduced imagegeneration unit with template images and to transmit comparison results;a specification unit operable to perform specification of the image areain accordance with the comparison results transmitted by the comparisonunit; and a control unit that, for each photographed image that isinput, is operable to (i) select M (M≦K) or fewer ratios for eachphotographed image in accordance with ratios indicated for aphotographed image input prior to each photographed image, (ii) indicatethe selected M or fewer ratios to the reduced image generation unit andmake the reduced image generation unit generate each reduced image,(iii) make the comparison unit compare each generated reduced image, and(iv) make the specification unit specify the image area for eachphotographed image in accordance with, at least, comparison resultstransmitted by the comparison unit.

Advantageous Effects of Invention

With the above-described structure, the image processing apparatusgenerates reduced images for each photographed image at M or fewerratios from among L ratios, in accordance with the processing capabilityof the apparatus (the capability to generate reduced images at K ratioswithin the predetermined time interval) and compares each reduced imagewith template images, thereby increasing the probability of detecting animage corresponding to a template within the predetermined time.

The control unit may alternately indicate to the reduced imagegeneration unit, as the M or fewer ratios for each photographed imagethat is input, (i) M or fewer ratios that include N (N≦M) ratiosselected from among odd numbered ratios and (ii) M or fewer ratios thatinclude N (N≦M) ratios selected from among even numbered ratios, theeven and odd numbered ratios corresponding to the L ratios arranged inascending or descending order.

With the above-described structure, the image processing apparatusalternately uses, for comparison with template images, reduced imagesgenerated at even numbered ratios and reduced images generated at oddnumbered ratios, the even and odd numbered ratios corresponding to the Lratios arranged in ascending or descending order. As compared to when Mor fewer ratios are selected arbitrarily from among L ratios, thisincreases the probability of detecting, in each photographed image,images of various sizes corresponding to template images.

The control unit may (i) select one or more ratios from among M or fewerratios indicated for a photographed image input prior to a particularphotographed image, in accordance with comparison results transmittedfrom the comparison unit for the photographed image input prior to theparticular photographed image, the comparison results indicating that animage corresponding to a template image was detected, and (ii) includethe selected ratios in the M or fewer ratios for the particularphotographed image which are indicated to the reduced image generationunit.

With the above-described structure, the image processing apparatusselects, from among the ratios used for generating the reduced imagesfor a photographed image input prior to a current photographed image,ratios for images detected as corresponding to a template image, andthus the apparatus generates reduced images for the current photographedimage using ratios from a prior detection. Therefore, when the currentphotographed image resembles the photographed image input prior to thecurrent photographed image, the probability of detecting, in the currentphotographed image, an image corresponding to a template image isincreased even further.

Examples of when the current photographed image would resemble thephotographed image input prior to the current photographed image includewhen the predetermined time interval is sufficiently short (e.g. 33 ms)or when photographed subjects are still.

The control unit may (i) select one or more ratios from among M or fewerratios indicated for a photographed image input prior to a particularphotographed image, in accordance with comparison results transmittedfrom the comparison unit for the photographed image input prior to theparticular photographed image, the comparison results indicating that animage corresponding to a template image was detected, and (ii) include,in the N ratios for the particular photographed image, ratios that areimmediately before or after the selected ratios in the ascending ordescending order of the L ratios.

With the above-described structure, the image processing apparatusselects, from among the ratios used for generating the reduced imagesfor a photographed image input prior to a current photographed image,ratios for images detected as corresponding to a template image, and theapparatus then generates reduced images for the current photographedimage using ratios in the vicinity of the selected ratios. Therefore,when the current photographed image resembles the prior inputphotographed image, the probability of detecting, in the currentphotographed image, an image corresponding to a template image isincreased even further.

This is because, when the current photographed image resembles thephotographed image input prior to the current photographed image, thereis a high probability of detecting, in the current photographed image,an image corresponding to a template image when using ratios in thevicinity of the ratios for which images were detected in thephotographed image input prior to the current photographed image.

The comparison unit may transmit, as the comparison results, pieces ofarea information each indicating an image area that is detected throughcomparison of each reduced image with template images, the image areacorresponding to one of the template images, and the specification unitmay perform the specification in accordance with (i) the pieces of areainformation for a particular photographed image and (ii) the pieces ofarea information for a photographed image input prior to the particularphotographed image, the pieces of area information in (i) and (ii) beingtransmitted by the comparison unit.

With the above-described structure, the image processing apparatusspecifies the image area for a current photographed image in accordancewith not only the pieces of area information for the currentphotographed image but also the pieces of area information for aphotographed image input prior to the current photographed image; thepieces of area information each indicating an image area that isdetected through comparison of each reduced image with template images,the reduced images being generated at M or fewer ratios from among Lpredetermined ratios. Therefore, the apparatus is able to suppress theeffects of error and increase the probability of specifying the imagearea.

The specification unit may specify one image area, in accordance withpieces of area information that indicate image areas that are positionedwithin a predetermined proximity to each other, from among (i) thepieces of area information for the particular photographed image and(ii) the pieces of area information for the photographed image inputprior to the particular photographed image, the pieces of areainformation in (i) and (ii) being transmitted by the comparison unit.

With the above-described structure, the image processing apparatusspecifies the image area for a current photographed image in accordancewith the pieces of information for the current photographed image andfor the photographed image input prior to the current photographed imagethat indicate image areas that are positioned in proximity to eachother, thereby suppressing the effects of error and increasing theprobability of specifying the image area.

The control unit may determine the M or fewer ratios for a particularphotographed image to indicate to the reduced image generation unit inaccordance with comparison results, transmitted by the comparison unit,for a photographed image input prior to the particular photographedimage.

With the above-described structure, if the image processing apparatusselects, for example, from among the ratios used for generating thereduced images for a photographed image input prior to a currentphotographed image, ratios for images detected as corresponding to atemplate image, then the apparatus generates reduced images for thecurrent photographed image using ratios from a prior detection.Therefore, when the current photographed image resembles thephotographed image input prior to the current photographed image, theprobability of detecting, in the current photographed image, an imagecorresponding to a template image is increased even further.

In the image processing apparatus, a plurality of candidate templateimages may be stored as candidates for the template images, and thecomparison unit may select the template images to use for comparison ofa particular photographed image from among the candidate templateimages, in accordance with comparison results for a photographed imageinput prior to the particular photographed image.

With the above-described structure, for example, if the image processingapparatus detects, in a photographed image input prior to the currentphotographed image, an image corresponding to a template image, then byusing that template image for comparison in the current photographedimage, it is possible to increase the probability of detecting, in ashorter amount of time, an image corresponding to a template image inthe current photographed image when the current photographed imageresembles the photographed image input prior to the current photographedimage.

The comparison unit may transmit, as the comparison results, pieces ofarea information each indicating an image area that is detected throughcomparison of each reduced image with template images, the image areacorresponding to one of the template images, and the specification unitmay perform the specification in accordance with (i) the pieces of areainformation for a particular photographed image and (ii) the pieces ofarea information for a photographed image input prior to the particularphotographed image, the pieces of area information in (i) and (ii) beingtransmitted by the comparison unit.

With the above-described structure, the image processing apparatusspecifies the image area for a current photographed image in accordancewith not only the pieces of area information for the currentphotographed image but also the pieces of area information for aphotographed image input prior to the current photographed image, thepieces of area information each indicating an image area that isdetected through comparison of each reduced image with template images,the reduced images being generated at M or fewer ratios from among Lpredetermined ratios. Therefore, the apparatus is able to suppress theeffects of error and increase the probability of specifying the imagearea.

The specification unit may specify one image area, in accordance withpieces of area information that indicate image areas that are positionedwithin a predetermined proximity to each other, from among (i) thepieces of area information for the particular photographed image and(ii) the pieces of area information for the photographed image inputprior to the particular photographed image, the pieces of areainformation in (i) and (ii) being transmitted by the comparison unit.

With the above-described structure, the image processing apparatusspecifies the image area for a current photographed image in accordancewith the pieces of information for the current photographed image andfor the photographed image input prior to the current photographed imagethat indicate image areas that are positioned in proximity to eachother, thereby suppressing the effects of error and increasing theprobability of specifying the image area.

The image processing apparatus may further comprise a display unit that,for each photographed image that is input, is operable to display both(i) the photographed image and (ii) a frame indicating an image area ata position on the photographed image corresponding to the image area asspecified for the photographed image by the specification unit.

With the above-described structure, the image processing apparatusdisplays a frame at the position of the image area specified for theinput photographed image, allowing the user to easily recognize theimage area that is the target of predetermined processing.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional block diagram of an image processing apparatus100.

FIG. 2 illustrates a face detection method used by a face detectionapparatus 180.

FIG. 3 shows an example of correlations between reduction IDs andreduction ratios.

FIG. 4 illustrates the arrangement of a photographed image 1000 in animage memory 130.

FIG. 5 is a flowchart showing the operations of the image processingapparatus 100.

FIG. 6 is a flowchart showing the specification processing of a facialimage area by the specification circuit 184.

FIG. 7 illustrates how the image processing apparatus 100 specifies afacial image area over time.

FIG. 8 illustrates how a conventional image processing apparatusspecifies a facial image area over time.

FIG. 9 is a flowchart showing the operations of the image processingapparatus in the modification.

FIG. 10 shows an example of the structure of a semiconductor integratedcircuit 200 that includes the face detection apparatus 180 described inthe embodiment.

DESCRIPTION OF EMBODIMENTS

The following is an explanation of an embodiment of the image processingapparatus in the present invention.

Embodiment 1

<Overview>

The image processing apparatus in the present embodiment specifies animage area that includes a facial image (hereinafter “facial imagearea”) in each photographed image generated by taking photographs at apredetermined interval (e.g. every 33 ms).

The photographed images are 4VGA size (1280×960 pixels), a relativelylarge size, and thus the image processing apparatus performs detectionprocessing of facial images by focusing on one region within the image(hereinafter, “detection target image”). The detection target image isan image with an area in which some sort of object was detected by aconventional edge detection method or the like.

This image processing apparatus detects a facial image by performingprocessing on the detection target image repeatedly at a predeterminedsize. In other words, the image processing apparatus generatessuccessively reduced images from this image of a predetermined size(hereinafter “comparison target image”) and compares each of thegenerated reduced images with a template image of a facial image todetect a facial image.

It is possible to use 12 different reduction ratios whose amount ofreduction is set to be successively larger as the reduction ratios forgenerating each reduced image. In this image processing apparatus,however, for each photographed image, rather than using all 12 reductionratios, six of these reduction ratios are used to generate reducedimages.

In the present embodiment, the image processing apparatus is assumed tohave the processing capability for generating reduced images at sixreduction rates within a predetermined amount of time.

This image processing apparatus specifies the facial image area by usingthe results of comparison with the template image for two photographedimages photographed consecutively.

In this way, as the image processing apparatus generates reduced imagesat six of the reduction rates among the 12 reduction rates, theprocessing load for generating and comparing the reduced images for eachphotographed image can be reduced, and furthermore as the apparatusspecifies the facial image area by using the comparison results for twophotographed images consecutively photographed, the facial image areacan be specified without a great loss in precision as compared toprocessing that generates and compares 12 different reduced images.

<Structure>

First, the following is a description of the structure of the imageprocessing apparatus 100.

FIG. 1 is a functional block diagram of the image processing apparatus100.

As shown in FIG. 1, the image processing apparatus 100 comprises acamera 110, a camera input circuit 120, an image memory 130, an LCD(Liquid Crystal Display) 140, an LCD output circuit 150, an objectdetection circuit 160, a processor 170, and a face detection apparatus180.

Data transfer between the processor 170 and the other constituentelements is performed via a processor bus 101. Also, access to the imagememory 130 from the other constituent elements besides the camera 110 isperformed via a memory bus 102.

The camera 110 captures images at a predetermined frame rate (e.g. 30fps (frames per second)) and transmits a sequentially generated 4VGAsize photographed image (data) to the camera input circuit 120.

The camera input circuit 120 performs, on the photographed imagereceived from the camera 110, various types of filter processing forimproving image quality and stores the filtered photographed image inthe image memory 130. The camera input circuit 120 notifies theprocessor 170 once it has stored the filtered photographed image in theimage memory 130.

The image memory 130 is a memory area for storing a filteredphotographed image.

The LCD 140 includes a liquid crystal display (LCD) and displays imagesin accordance with instructions from the LCD output circuit 150.

At the same frame rate as the frame rate for photography, the LCD outputcircuit 150 reads the photographed image stored in the image memory 130and causes the LCD 140 to display the image, while also, in accordancewith instructions from the processor 170, causing the LCD 140 to displaya rectangular frame indicating the specified facial image area.

In accordance with instructions from the processor 170, the objectdetection circuit 160 reads the photographed image recorded in the imagememory 130, performs object detection processing, and transmits thedetection results, which include information indicating whether anobject was detected, to the processor 170.

This object detection is performed, for example, by comparing edgedetection results for (i) a photographed image in which no object isshown and (ii) the photographed image recorded in the image memory 130.In particular, when an object is detected, detection results thatinclude the coordinate values (for example, the coordinate values of theupper left and lower right edges) of the rectangular detection targetimage, which includes the detected object, are transmitted to theprocessor 170.

By executing a control program stored in internal memory (not shown),the processor 170 controls the image processing apparatus 100 as a wholeand provides instructions and notifications to each block in theapparatus.

In particular, when the processor 170 receives detection results fromthe object detection circuit 160 that include information indicatingthat an object was detected, it notifies the face detection apparatus180 of the coordinate values for the detection target image (included inthe detection results) and of the reduction ratios for generating thereduced images. When the processor 170 receives detection results fromthe object detection circuit 160 that include information indicatingthat an object was not detected, it notifies the face detectionapparatus 180 that no object was detected.

The processor 170 also causes the LCD output circuit 150 to display arectangular frame indicating the specified facial image area inaccordance with the coordinate values or the like indicating thespecified facial image area.

The face detection apparatus 180 processes the detection target imageindicated by the coordinate values of which it was notified by theprocessor 170 in units of a predetermined size (QVGA size (320×240pixels)) and specifies a facial image area. More concretely, the facedetection apparatus 180 detects a facial image by comparing reducedimages, which are a reduction of this image of a predetermined size (thecomparison target image), with each template image, and based on thecomparison results, it specifies a facial image area.

The face in each template image faces a different direction, and byusing each template image for comparison, facial images that face avariety of directions in the reduced image can be detected.

The face detection apparatus 180 comprises a result memory 181, aresolution conversion circuit 182, a face detection circuit 183, and aspecification circuit 184.

The result memory 181 is a memory area for storing (i) a list(hereinafter “result list”) of each of the comparison results for thephotographed image being processed by the face detection apparatus 180and (ii) the result list for the photographed image taken one imageprior to the current photographed image.

The resolution conversion circuit 182 determines, inside the detectiontarget image indicated by the coordinate values of which it was notifiedby the processor 170, the comparison target images (QVGA size) fromwhich reduced images are to be generated. The resolution conversioncircuit 182 then generates reduced images of the determined comparisontarget images, using the reduction ratios of which the processor 170provided notification, and transmits the reduced images to the facedetection circuit 183.

The comparison target images can be determined using a similar method tothe method disclosed in PCT/JP2008/000961, which establishes overlapareas from the upper left to the lower right of the detection targetimage. Therefore, details are omitted here.

In the present embodiment, the reduction ratio is the ratio of the sizeof the comparison target image (QVGA size), set as 1, to the size of theimage after reduction. Therefore, when the reduction ratio is large (forexample, 0.82), the amount of reduction is small, and conversely whenthe reduction ratio is small (for example, 0.11), the amount ofreduction is large.

In the present embodiment, even when the reduction ratio is 1.0, i.e.when the reduced image is identical to the comparison target image, theterm “reduced image” is still used.

By comparing the reduced image input from the resolution conversioncircuit 182 with each template image (each of a predetermined size of24×24 pixels), the face detection circuit 183 detects a facial imageincluded in the reduced image.

More concretely, for each template image, the face detection circuit 183performs processing that compares the reduced image with the templateimage while shifting the template image from the upper left pixel to thelower right pixel in the received reduced image, storing each of thecomparison results in the result memory 181 as the result list for thephotographed image being processed by the face detection apparatus 180.

In addition to information indicating whether a facial image wasdetected or not, in particular when an object was detected, thecomparison results include (i) coordinate values which are the result oftransformation into coordinates on the photographed image of thecoordinate values in the reduced image indicating the upper left edge ofthe detected rectangular facial image, (ii) information indicating thereduction ratio (hereinafter, “reduction ID”) for the reduced image thatincludes the detected facial image, and (iii) information indicating thedirection faced by the face in the template (hereinafter, “template ID”)used for detection.

In the example below, the face detection circuit 183 uses three templateimages for comparison: a face that faces forward, a face turned to theright, and a face turned to the left. The template IDs for the templateimages are, respectively, “0,” “1,” and “2.”

Since the reduced image and the photographed image are similar, theabove-mentioned transformation from the coordinate values in the reducedimage to coordinate values in the photographed image can be performed bysimple comparative calculation using the ratio of the horizontal sizesand vertical sizes of both images, and thus a detailed explanation isomitted.

The specification circuit 184 specifies a facial image area based on thecomparison results included in the result lists, stored in the resultmemory 181, for the photographed image currently being processed by theface detection apparatus 180 and for the photographed image taken oneimage prior. The specification circuit 184 then transmits coordinatevalues and other information indicating the determined facial image areato the processor 170. Details regarding the method for specification ofthe facial image area are provided below.

<Face Detection>

Next, the following is a description of the face detection method usedby the face detection apparatus 180.

FIG. 2 illustrates the face detection method used by the face detectionapparatus 180.

In this figure, a comparison target image 1021 and a comparison targetimage 1022 each include a facial image, and the facial image included inthe comparison target image 1021 is larger than the facial imageincluded in the comparison target image 1022.

In order to compare the comparison target image 1021 with a templateimage of a predetermined size (for example 24×24 pixels) and detect afacial image, the size of the facial image included in the comparisontarget image 1021 must be reduced to the same extent as the size of thefacial image in the template image. A reduced image 1031 refers to theimage that has been reduced in this way.

The face detection circuit 183 detects a facial image by repeatedlycomparing the template image with the reduced image while shifting thetemplate image over the reduced image 1031 from the top left pixel tothe bottom right pixel.

Detection is performed similarly when detecting a facial image in thecomparison target image 1022: the face detection circuit 183 repeatedlycompares a reduced image 1032 that is a reduction of the comparisontarget image 1022 to the template image.

In this example, the size of the reduced image 1032 is larger than thesize of the reduced image 1031. This indicates that the reduced image1032 has been reduced to a lesser extent than the reduced image 1031.Since the size of the facial image included in the comparison targetimage 1022 is smaller than the size of the facial image included in thecomparison target image 1021, the size of the facial image included inthe target image 1022 can thus be reduced to the same degree of size asthe facial image in the template image at a lower degree of reduction.

To detect the facial image in the comparison target image in this way,it is necessary to compare the template image with a reduced image thathas been reduced at a reduction ratio in accordance with the size of thefacial image included in the comparison target image. However, since thesize of the facial image included in the comparison target image is notknown when actually attempting to detect the facial image, the facedetection circuit 183 detects the facial image by repeatedly comparingthe template image to the reduced image while gradually changing thereduction ratio of the reduced image.

As shown in FIG. 3, for example, each reduction ratio for the reducedimage can be determined by incrementally changing the reduction ratioused previously by a factor of 1/1.22.

FIG. 3 shows an example of correlations between reduction IDs andreduction ratios.

In this figure, the notation (1/1.22) ̂n indicates (1/1.22) raised tothe n^(th) power.

In this figure, for example, the reduction ratio when the reduction IDis “0” is (1/1.22) ̂0, that is, 1.0, and when the comparison targetimage is reduced by the corresponding reduction ratio, the resultingreduced image has a horizontal size of 320 pixels and a vertical size of240 pixels.

When, on the other hand, the reduction ID is “11,” the reduction ratiois (1/1.22) ̂11, the horizontal size of the reduced image is 33 pixels,and the vertical size is 25 pixels.

Note that since matching cannot be performed when the size of thereduced image is smaller than the size of the template image (in theabove example, 24×24 pixels), in this example, the smallest reductionratio is “11,” i.e. this reduction ratio indicates the greatest degreeof reduction.

<Data>

Next, the following is a description of the arrangement of aphotographed image in the image memory 130.

FIG. 4 illustrates the arrangement of a photographed image 1000 in animage memory 130.

As described above, the photographed image 1000 is constituted from4VGA, size (1280×960 pixels) image data, and pixel value data for thepixels constituting the photographed image 1000 are stored sequentiallyin continuous regions of the image memory 130. For example, the pixelsshown in FIG. 4 are stored in continuous regions in the following order:1001, . . . , 1002, 1003, . . . , 1004, . . . , 1005.

In this example, when an address value of an area in the image memory130 in which a pixel 1001 is stored is expressed as BASE, address valuesof areas in which the pixel data values are stored are calculated withuse of the following formula. Note that the horizontal size in theformula is the horizontal size of the photographed image 1000, that is1280 pixels. The coordinate value of the pixel 1001 is defined as (0,0), with the X axis extending to the right, and the Y axis extendingdown. Also, one address indicates an area in the image memory 130 whereone pixel worth of image data is stored.

Address value=BASE+horizontal size×Y axis+X axis   Equation 1

For example, the address value of an area in the image memory 130 inwhich a pixel 1004 with coordinate values (640, 240) is stored is(BASE+1280×240+640).

The resolution conversion circuit 182 in the face detection apparatus180 can read a determined comparison target image from the image memory130 by using the address values calculated from the coordinate values ofthe comparison target image.

<Operation>

Next, the operation of the image processing apparatus 100, which handlesthe above-described data and has the above-described structure, isdescribed.

FIG. 5 is a flowchart showing the operations of the image processingapparatus 100.

Upon receiving notification from the camera input circuit 120 that aphotographed image has been stored in the image memory 130, theprocessor 170 increments the photographed image counter and causes theobject detection circuit 160 to begin detection processing. Once it hasreceived the detection results from the object detection circuit 160,the processor 170 determines whether the detection results includeinformation indicating that an object was detected (step S1).

If information indicating that an object was detected is included (stepS1: YES), then the processor 170 determines whether the photographedimage in which an object was detected is sequentially an even numberedimage or not in accordance with the photographed image counter (stepS2). For example, if the photographed image counter is 1, then thedetermination is negative, whereas if the photographed image counter is2, the determination is positive.

When the photographed image in which an object was detected issequentially an even numbered image (step S2: YES), then the processor170 selects the even numbered reduction IDs (0, 2, 4 . . . 10) fromamong the 12 reduction IDs (step S3), and when the photographed image inwhich an object was detected is sequentially an odd numbered image (stepS2: NO), then the processor 170 selects the odd numbered reduction IDs(1, 3, 5 . . . 11) (step S4).

The processor 170 notifies the face detection apparatus 180 of theselected reduction IDs and of the coordinate values of the detectiontarget image included in the input detection results (step S5).

The resolution conversion circuit 182 in the face detection apparatus180 selects one comparison target image in the detection target imageindicated by the coordinate values of which the processor 170 providednotification (step S6). In accordance with Equation 1, the resolutionconversion circuit 182 then calculates the address value in the imagememory 130 in which the selected comparison target image is stored andreads the comparison target image from the image memory 130 (step S7).

The resolution conversion circuit 182 selects one reduction ID fromamong the reduction IDs of which it was notified in step S5 andgenerates, at the reduction ratio corresponding to the selectedreduction ID, a reduced image for the comparison target image which ithas read, sending the reduced image to the face detection circuit 183(step S8). The resolution conversion circuit 182 may select, forexample, the smallest reduction ID from among the reduction IDs whichhave not yet been selected.

In the first iteration, the face detection circuit 183 generates anempty result list in the result memory 181 for the photographed imagebeing processed by the face detection apparatus 180. Then, the facedetection circuit 183 compares the reduced image input from theresolution conversion circuit 182 with each template image whileshifting the template image from the upper left pixel to the lower rightpixel in the reduced image, recording the comparison results in theresult list (step S9).

The resolution conversion circuit 182 determines whether all of thereduction IDs of which it was notified in step S5 have been selected ornot (step S10), and when not all reduction IDs have been selected (stepS10: NO), the processing from step S7 on is performed again so as togenerate reduced images for the reduction IDs that have not yet beenselected.

If, on the other hand, all of the reduction IDs have been selected (stepS10: YES), then the resolution conversion circuit 182 determines whetherprocessing has been performed for the whole detection target image ornot (step S11). When processing has not been performed for the wholedetection target image (step S11: NO), then the processing from step S6on is performed again so as to process the next comparison target image.

If in step 11 processing has been performed for the whole detectiontarget image (step S11: YES), then the specification circuit 184performs specification processing on the facial image area in accordancewith each of the comparison results included in two result lists storedin the result memory 181 (step S12).

If, on the other hand, information indicating that an object was notdetected is included in the detection results input from the objectdetection circuit 160 (step S1: NO), then the processor 170 notifies theface detection apparatus 180 that an object was not detected. The facedetection circuit 183 also generates a empty result list in the resultmemory 181 for the photographed image being processed by the facedetection apparatus 180, and the specification circuit 184 performsspecification processing on the facial image area in the same way asabove (step S12).

Based on the information for the specified facial image area transmittedby the specification circuit 184 as the results of the processing instep S12, the processor 170 notifies the LCD output circuit 150 of thecoordinate values (coordinate values of the upper left and lower rightedges) of the facial image area to be drawn on top of the photographedimage. The LCD output circuit 150 draws a rectangle, indicating thefacial image area of which it was notified by the processor 170, on topof the photographed image read from the image memory 130 and causes theLCD 140 to display the photographed image and the rectangle, which endsthe processing sequence. If, as a result of the processing in step S12,the processor 170 receives notification from the specification circuit184 that a facial image area could not be specified, then it is possiblenot to have the processor 170 notify the LCD output circuit 150 of theabove-described coordinates, or to have the processor 170 notify the LCDoutput circuit 150 of the above-described coordinates based oninformation for the nearest facial image area specified by thespecification circuit 184.

The following is an explanation of the specification processing for thefacial image area in step S12.

FIG. 6 is a flowchart showing the specification processing of a facialimage area by the specification circuit 184.

For each of the comparison results included in two result lists storedin the result memory 181, the specification circuit 184 calculates thecenter points for detected facial images based on the coordinate valuesof the upper left edge of the detected facial image (coordinate valueson the photographed image) and on the reduction ID, which are includedin the comparison results (step S13).

For example, if the coordinate values for the upper left edge of thedetected facial image are (92, 48) and the reduction ID is “2,” thenbased on the size of the template image (24×24 pixels) and on thereduction ratio corresponding to the reduction ID ((1/1.22) ̂2), thesize of the facial image can be calculated to be 36×36 pixels.Therefore, the center point of the detected facial image can becalculated as (92+36/2, 48+36/2), or (110, 66).

If no object was detected by the object detection circuit 160 (step S1:NO), then as described above, an empty result list is generated thatincludes no comparison results. Furthermore, when no facial image isdetected by the face detection circuit 183, then no coordinate valuesare included in the comparison results in the result list. Therefore, insome cases the center points may be calculated for comparison resultsincluded in only one result list.

Additionally, while not shown in the figures, when coordinate values ofthe upper left edge of the detected facial image are not included ineither result list, the specification circuit 184 notifies the processor170 that the facial image area could not be specified, and specificationprocessing ends.

The specification circuit 184 determines whether a group has not beendetermined for any of the calculated center points (step S14). If thereis one or more center point whose group has not been determined (stepS14: YES), then the specification circuit 184 selects one of the centerpoints whose group has not been determined (step S15). Hereinafter, thiscenter point is referred to as the “selected point.” The specificationcircuit 184 can be made to select, for example, a center pointcorresponding to coordinate values included in the comparison resultsthat indicate the smallest reduction ID.

The specification circuit 184 chooses, from among the center pointswhose group has not been determined, center points whose reduction ID iswithin a predetermined range (e.g. within a range of ±2) of the selectedpoint' s reduction ID as candidates for center points belonging to thesame group as the selected point's group (step S16). Hereinafter, thesecenterpoints chosen as candidates are referred to as “candidate points.”

For example, if the reduction ID for the selected point is “2” thencenter points with a reduction ID between “038 and “4” are candidatepoints.

The specification circuit 184 calculates the distance between theselected point and each candidate point (step S17), and candidate pointsfor which the calculated distance is within a predetermined value (e.g.20) are determined to be in the same group as the selected point (stepS18).

For example, if the coordinate values of a selected point are (110, 66)and the coordinate values of a candidate point are (188, 68), then thedistance between the points is “8.2,” and the candidate point isdetermined to belong to the same group as the selected point.

Based on the coordinate values and the size of each facial image foreach center point belonging to the same group, the specification circuit184 specifies a facial image area and transmits the coordinate values ofthe upper left edge of the facial image area as well as the size of thearea to the processor 170 (step S19).

As a concrete example, if the two center points mentioned above, (110,66) and (118, 68) are determined to be in the same group, then the meanvalue for the coordinates of these center points is (114, 67).

Furthermore, the size of the facial image with a center point of (110,66) is 36×36 pixels, as described above. If, for example, the size ofthe facial image with a center point of (118, 68) is 44×44 pixels, thenthe mean value for the size of these images is 40×40 pixels.

The specification circuit 184 specifies as a facial image area the imagearea whose center point is the calculated mean value (114, 67) of thecoordinates of the center points and whose size is the calculated meanvalue (40×40 pixels) of the sizes of the images. The specificationcircuit 184 then transmits the coordinate values for the upper left edgeof the specified facial image area, (114−40/2, 67−40/2), i.e. (94, 47),as well as the size (40×40 pixels) to the processor 170.

When the processing in step S19 is complete, the specification circuit184 once again performs processing starting with step S14. When there isno center point whose group has not been determined (step S14: NO), thenthe result list for the photographed image taken one image prior to thephotographed image currently being processed by the face detection unit180 is deleted from the result memory 181, and specification processingfor the facial image area is complete.

<Discussion>

The following is an explanation, with reference to FIGS. 7 and 8, ofboth the image processing apparatus 100 and a conventional imageprocessing apparatus as they specify a facial image area.

<Image Processing Apparatus 100>

First, the image processing apparatus 100 is explained.

FIG. 7 illustrates how the image processing apparatus 100 specifies afacial image area over time.

This figure shows a row of photographed images 1011-1013 captured andgenerated by the camera 100 at 33 ms intervals.

The photographed image counter for the photographed images 1011-1013 isdefined as n to n+2, with n being an even number.

The displayed images are the row of images 1041-1043 displayed by theLCD 140 at 33 ms intervals.

First, at T1, the photographed image 1011 is generated, and since inthis example the photographed image counter for the photographed image1011 is an even number, between T1 and T2, reduced images are generatedusing even numbered reduction IDs (0, 2 . . . 10), and comparisonprocessing with each template is performed.

Next, at T2, when the photographed image 1012 is generated, reducedimages are produced using odd numbered reduction IDs (1, 3 . . . 11) andcomparison processing is performed, as with the photographed image 1011.At the same time, specification processing of the facial image area inthe photographed image 1011 is performed.

This specification of the facial image area is performed based on thecomparison results for the photographed image 1011 and the comparisonresults of the photographed image taken one image prior to thephotographed image 1011. A rectangular frame S1 indicating the specifiedfacial image area is drawn on top of the photographed image 1011(displayed image 1041) and is displayed continually until T3.

Note that, on top of the displayed image 1041, dotted-line ellipses C0and C1, as well as dotted-line rectangles are shown, but these shapesare only for the sake of explanation and are not actually displayed. Thesame holds true for displayed images 1042 and 1043.

The dotted-line rectangles indicate each detected facial image. Thedotted-line ellipse C0 indicates the area that includes each facialimage detected in the photographed image taken one image prior to thephotographed image 1011, and the dotted-line ellipse C1 indicates thearea that includes each facial image detected in the photographed image1011.

In other words, specification of the facial image area for thephotographed image 1011 is performed based on the facial images includedin the ellipse C1 and the facial images included in the ellipse C0.

Next, at T3, when the photographed image 1013 is generated, using evennumbered reduction IDs (0, 2 . . . 10), reduced images are produced andcomparison processing is performed for this photographed image, as withthe photographed image 1011 at T1. Specification processing of thefacial image area is performed for the photographed image 1012 as withthe photographed image 1011 at T2, and a rectangular frame S2 indicatingthe specified facial image area is drawn on top of the photographedimage 1012 (displayed image 1042). The frame is displayed continuallyuntil T4.

Specification of the facial image area in the photographed image 1012 isperformed based on each facial image (the facial images included inellipse C2) detected in the photographed image 1012 and on each facialimage (the facial images included in ellipse C1) detected in thephotographed image 1011.

The photographed image 1013 is then processed in the same way.

It is therefore clear that the rectangular frames S1-S3 thus displayedon the displayed images 1041-1043 are displayed near the facial imagethat is actually shown.

<Conventional Image Processing Apparatus>

The following is an explanation of a conventional image processingapparatus, which generates reduced images using all 12 reduction ratesand compares each reduction image with each template image.

FIG. 8 illustrates how a conventional image processing apparatusspecifies a facial image area over time.

As in the explanation of FIG. 7, the photographed images 2011-2015 inthis figure are captured and generated at 33 ms intervals. Thephotographed images 2011-2013 are exactly the same as the photographedimages 1011-1013 in FIG. 7. Furthermore, the photographed image counterfor the photographed images is defined as n to n+4.

The displayed images are the images 2021-2025 displayed at 33 msintervals.

In the following explanation, the time required for the conventionalimage processing apparatus to complete reduction processing andcomparison processing for a single photographed image is 66 ms.

First, at T1, the photographed image 2011 is generated, and by T3,reduced images are generated using all of the reduction IDs (0, 1 . . .11) and comparison processing with each template is performed.

Next, at T2, since comparison with all of the reduced images for thephotographed image 2011 has not been completed, the photographed image2011 is displayed as is (displayed image 2021). Also, at T2, thephotographed image 2012 is generated, but since comparison with all ofthe reduced images for the photographed image 2011 has not beencompleted, reduction processing and comparison processing are notperformed for the photographed image 2012 at this point.

Next, at T3, based on the comparison results from T1-T3 for thephotographed image 2011, specification processing of the facial imagearea in the photographed image 2011 is performed. A rectangular frameS10 indicating the specified facial image area is drawn on top of thephotographed image 2012 (displayed image 2022) and is displayedcontinually until T4. Note that this conventional image processingapparatus specifies the facial image area by calculating the mean valuesof the coordinate values and the sizes for each detected facial image.

At T3, the photographic image 2013 is generated, and using all reductionIDs (0, 1 . . . 11), reduction processing and comparison processing forthis image is performed by T5 as with the photographed image 2011 at T1.

Next, at T4, since comparison with all of the reduced images for thephotographed image 2013 has not been completed, the rectangular frameS10 indicating the specified facial image area for the photographedimage 2011 is drawn as is on top of the photographed image 2013(displayed image 2023) and is displayed continually until T5.

In the same way, a rectangular frame S11 indicating the detected facialimage area for the photographed image 2013 is later drawn on top of boththe photographed images 2014 and 2015 (displayed images 2024, 2025). Theframe is displayed from T5 to T6 for the displayed image 2024 and fromT6 to T7 for the displayed image 2025.

In this way, in a conventional image processing apparatus, by generatingreduced images for all of the reduction IDs and comparing them with eachtemplate image, the amount of time required for reduction processing andcomparison processing ends up being longer than the time intervalbetween photographs (in this example, 33 ms), and detection accuracy forfacial images actually grows worse. In particular, displaying therectangular frame that indicates the previous specified facial imagearea in displayed images 2023 and 2025 causes a great misalignmentbetween the position of the facial image actually shown and the positionof the rectangular frame.

As the image processing apparatus 100 in the embodiment of the presentinvention, in accordance with the processing capabilities of theapparatus, generates reduced images by alternately using reductionratios corresponding to six even numbered reduction IDs and reductionratios corresponding to six odd numbered reduction IDs from among the 12reduction ratios, the apparatus is thus capable of performing reductionprocessing and comparison processing for each photographed image withinthe time interval between photographs (in this example, 33 ms), avoidingthe problems of a conventional image processing apparatus.

Furthermore, since the facial image area for each photographed image isspecified using the comparison results for both, the photographed imageand for the photographed image taken one image prior, the effects oferror can be reduced, allowing the facial image area to be specifiedaccurately.

<<Modifications>>

The following is an explanation of a modification in which the methodfor selecting the reduction ratios which the image processing apparatususes for generating reduced images is modified.

Since the image processing apparatus in the modification (hereinafter“modified image processing apparatus”) is a slight modification of theabove-described functions of the processor 170 in the image processingapparatus 100 and of the face detection circuit 183, only the modifiedelements are described below.

<Operations>

The following is an explanation of the operations of the modified imageprocessing apparatus.

FIG. 9 is a flowchart showing the operations of the modified imageprocessing apparatus.

As shown in the figure, the modified image processing apparatus differsfrom the image processing apparatus 100 in that it includes theprocessing in step S22 instead of the processing instep S9 in FIG. 5.

The processing in step S22 differs from the processing in step S9 inthat, in step S22, the face detection circuit in the modification notonly records the comparison results in the result list generated in theresult memory 181, but also transmits these comparison results to theprocessor in the modification.

Furthermore, the modified image processing apparatus differs from theimage processing apparatus 100 by including, in addition to eachprocessing step shown in FIG. 5, processing steps S20 and S21.

If information indicating that an object was detected is included in thedetection results input from the object detection circuit 160 (step S1:YES), then the processor in the modification determines whether or not afacial image was detected in the photographed image taken one imageprior to the photographed image currently being processed, in accordancewith the comparison results transmitted, in the above-described stepS22, by the face detection circuit in the modification (step S20). Inother words, a positive determination is only made when informationindicating that a facial image was detected is included in thesecomparison results. Note that when information indicating that a facialimage was not detected is included in the detection results input fromthe object detection circuit 160 for the photographed image taken oneimage prior, then a negative determination is made.

When a facial image has not been detected in the photographed imagetaken one image prior (step S20: NO), then the processing in step S2 isperformed, and in accordance with the processing results, the reductionIDs are selected in either step S3 or step S4.

When a facial image has been detected in the photographed image takenone image prior (step S20: YES), then six or fewer reduction IDs areselected from among (i) the reduction IDs included in the comparisonresults that include information indicating that a facial image wasdetected and (ii) reduction IDs in the vicinity of the reduction IDs in(i) (step S21).

For example, if the reduction ID included in the comparison resultswhich include information indicating that a facial image was detected is“2,” then the reduction IDs in the vicinity are “1” and “3.” Therefore,in this example the reduction IDs “1”-“3” are selected.

When the number of reduction IDs selected is fewer than six, additionalreduction IDs may be selected at will in order to bring the number ofreduction IDs up to six.

The reduction IDs selected in steps S21, S3, or S4, as well as thecoordinate values for the detection target image included in the inputdetection results are notified to the face detection apparatus in themodification (step S5). Subsequent processing is the same as theprocessing explained in FIG. 5.

<Supplementary Remarks>

This concludes the explanation of the image processing apparatus in thepresent invention based on an embodiment and a modification thereof. Thefollowing modifications, however, are also possible, and of course thepresent invention is not limited to an image processing apparatusexactly as shown in the embodiment or the modification described above.

(1) The structural elements described in the embodiment and themodification may be partially or completely realized as a computerprogram, or as an integrated circuit composed of one chip or a pluralityof chips.

FIG. 10 shows an example of the structure of a semiconductor integratedcircuit 200 that includes the face detection apparatus 180 described inthe embodiment.

As shown in FIG. 10, the semiconductor integrated circuit 200 includes asemiconductor integrated circuit corresponding to the face detectionapparatus 180, and depending on the use of the apparatus to which thesemiconductor integrated circuit 200 is applied; may also include a ROM210, an image encoding circuit 220, an audio processing unit 230, etc.

The, semiconductor integrated circuit 200 is generally structured as anMOS transistor such as a CMOS, and a specified logic circuit is realizedby a structure of connections with the MOS transistor. In recent years,the integration degree of semiconductor integrated circuits hasprogressed, and since an extremely complicated logic circuit can berealized by one or more semiconductor integrated circuits, the facedetection apparatus 180 can be made compact, and low energy consumptioncan be achieved.

Note that the semiconductor integrated circuit 200 has been described asincluding a semiconductor integrated circuit corresponding to the facedetection apparatus 180 in the embodiment. However, the semiconductorintegrated circuit 200 may alternatively include a semiconductorintegrated circuit corresponding to the face detection apparatus in themodification.

(2) Although in the embodiment and the modification, explanation was notprovided in particular for the case when a plurality of facial imageswere included in a photographed image, i.e. when a plurality ofdetection target images was detected. Processing may be performed,however, to detect facial images in each of a plurality of detectiontarget images.

(3) In the embodiment and the modification, the size of the photographedimage was described as 4VGA, the size of the comparison target image asQVGA, and the size of the template image as 24×24 pixels. The presentinvention is not limited to these sizes, however, and sizes may beselected arbitrarily.

(4) The template images for the embodiment and the modification weredescribed as being of three types, but the number of template images isnot limited to three. For example, only a template image that facesforward may be used, or template images for a larger number of facialangles than three may be used for comparison.

(5) The face detection apparatus in the embodiment and the modificationwas described as comparing a reduced image with each of a plurality oftemplate images, but the template images used for comparison may benarrowed down based on the comparison results in the result list, storedin the result memory 181, for the photographed image taken one imageprior.

In other words, the one or more template images indicated by the one ormore template IDs included in the comparison results that includeinformation indicating that a facial image was detected for thephotographed image one image prior may be used for comparison.

(6) In the embodiment and the modification, comparison is described asbeing performed in all comparison target areas in the detection targetarea regardless of whether a facial image has been detected. However,when according to a predetermined condition (for example, that a degreeof coincidence with the template image is greater than or equal to apredetermined threshold) a determination is made that a facial image hasbeen detected, the detection processing for that photographed image maybe ended.

(7) Although in the embodiment and the modification, the facial imagedetector is described as detecting a facial image, it may also detect aspecified image other than a face. The specified image may also be, forexample, an identification tag attached to a person or object, and inthis case, the specified image may be used to identify an individual ora type of object based on the detected identification tag.

(8) In the embodiment and modification, the object detection circuit 160is described as detecting an object by comparing edge detection resultsof a photographed image in which an object is not shown and aphotographed image stored in the image memory 130. This is only anexample, however, and detection may be performed by another method suchas by finding the difference between pixel values of successivelygenerated photographed images.

(9) In the embodiment and the modification, the reduction ratio isdescribed as changing by (1/1.22) ̂n each time as shown in FIG. 3. Thisis only one example, however, and the reduction ratios may be set tochange by another amount, for example (1/1.25) ̂2, as long as the sizeof the image that has been reduced by the smallest reduction ratio (thatis, the image having the greatest degree of reduction) is larger thansize of the template image.

(10) In the embodiment and the modification, an example of the imageprocessing apparatus was described in which the apparatus generatedreduces images for a selected comparison target image by successivelychanging the reduction ID, as shown in steps S6-S8 of FIGS. 5 and 9.Reduced images may be generated, however, for one reduction ID whilesuccessively changing the position of the comparison target image.

(11) In the embodiment and the modification, as shown in step S7 ofFIGS. 5 and 9, the resolution conversion circuit 182 was described asreading the comparison target image from the image memory 130 every timereduced images are generated, yet this circuit may store a generatedreduced image and use this stored image to generate reduced images atother reduction ratios. For example, if the resolution conversioncircuit 182 was notified by the processor 170 of even numbered reductionIDs, it may read the comparison target image once from the image memory130, generate a reduced image at the reduction ratio corresponding tothe reduction ID 0, and generate the following reduced imagescorresponding to reduction IDs 2, 4 . . . 10 by multiplying thegenerated reduction image by (1/1.22) ̂2.

(12) In the embodiment and modification, the operations of the imageprocessing apparatus as shown in steps S2-S4 in FIGS. 5 and 9 weredescribed as follows: when the photographed image counter for thephotographed image subject to detection is an even number (step S2:YES), the apparatus selects even numbered reduction IDs (step 53), andwhen the photographed image counter for the photographed image is an oddnumber (step S2: NO), the apparatus selects odd numbered reduction IDs(step 54).

However, such selection can be reversed. In other words, steps S3 and S4can be switched.

(13) The object detection circuit 160 and the processor in theembodiment and the modification have been described as not beingincluded in the face detection apparatus. The object detection circuit160 and the processor may, however, be included in whole or in part inthe face detection apparatus.

(14) In the embodiment and the modification, the selection processingfor a comparison target image shown in FIGS. 5 and 9 (step S6) wasdescribed as being performed by the resolution conversion circuit 182,but this processing may be performed by the processor, having theprocessor then notify the resolution conversion circuit 182 of thecoordinate values of the selected comparison target image.

(15) The processor in the embodiment and the modification was describedas making the object detection circuit 160 perform detection processingfor all of the photographed images. After the object detection circuit160 performs detection processing once, however, it can be made not toperform detection processing for a predetermined number of photographedimages, performing processing once again for the next photographed imageafter the predetermined number of images has been taken.

In this case, for each photographed image taken while the objectdetection circuit 160 is not performing detection processing, theprocessor notifies the face detection apparatus of the coordinate valuesfor the detection target image included in the preceding detectionresults received from the object detection circuit 160.

Furthermore, during this time, the processor may be made not to notifythe face detection apparatus of the coordinate values for the detectiontarget image, and the face detection apparatus may store the coordinatevalues for the detection target image of which it was notified once bythe processor until it is notified again by the processor of newcoordinate values for the detection target image, selecting a comparisontarget image within the comparison target images indicated by the storedcoordinate values.

When it has detected a facial image, the face detection circuit maystore the coordinate values of the comparison target images as convertedinto coordinates in the photographed image, and while the processor isnot providing notification of the coordinate values of the detectiontarget image, processing of the photographed image currently beingprocessed may only be performed for the comparison target imagesindicated by the stored coordinate values.

(16) The processor in the embodiment and the modification was describedas making the object detection circuit 160 perform detection processingon all of the photographed images, but after detection has beenperformed once, the object detection circuit 160 may be made not toperform detection processing until it is notified by the face detectionapparatus that a facial image area could not be specified. In themeantime, the same variation as described above in (15) can be used asthe selection method for each comparison target image for comparisonprocessing.

(17) In the embodiment and the modification, the specification circuit184 was described as transmitting the coordinate values for the upperleft edge of the specified facial image area as well as the size of theimage area to the processor, but if the facial image area can bedetermined, other information may be transmitted. For example, thecoordinate values for the lower right edge could be transmitted, ascould the coordinate values for both the upper left edge and the lowerright edge.

Furthermore, the processor was described as notifying the face detectionapparatus of the coordinate values for the upper left edge and lowerright edge of the detection target image, but if the detection targetimage can be specified, then similarly other information can benotified, such as the coordinate values for the upper left edge or thelower right edge along with the size of the image.

The same is also true for the coordinate values included in thedetection results transmitted to the processor by the object detectioncircuit 160, i.e. the coordinate values for the upper left edge andlower right edge of the detection target image.

(18) In the discussion of the embodiment, reduction processing andcomparison processing were described as being performed for thephotographed image 1011 between T1 and T2, with specification processingof the facial image area for the photographed image 1011 starting at T2,as shown in FIG. 7. When the image processing apparatus 100 hasprocessing capabilities to spare, however, the apparatus may be made toperform reduction processing, comparison processing, and specificationprocessing for one photographed image within the time interval betweenphotographs (in the above-described example, 33 ms).

Reduction processing, comparison processing, and specificationprocessing can also be performed within the time interval betweenphotographs by decreasing the number of reduced images that the imageprocessing apparatus 100 generates. Since reducing the number of reducedimages that are generated generally leads to less accuracy duringdetection of facial images, even when the image processing apparatus ismodified so that the number of reduced images that are generated isdecreased, it is assumed that accuracy will be within the rangepermitted by the image processing apparatus.

These changes can also be applied to the modified image processingapparatus.

(19) The image processing apparatus in the embodiment and themodification were described as generating reduced images for eachphotographed image using six or fewer reduction IDs from among 12reduction IDs. However, the user may be allowed to select between a modethat uses all 12 reduction IDs and a mode that uses six or fewerreduction IDs, with the apparatus generating a number of reduced imagescorresponding to the selected mode.

It is assumed that the image processing apparatus in the embodiment andthe modification does not have the processing capability to generate all12 reduced images within a predetermined time interval (in this example,33 ms). For subjects that are almost at rest, however, the problem of aspecified image area and the actual area in the subject becoming greatlyout of alignment will not occur, even if a specified image is detectedafter the predetermined time interval has passed. Therefore, the usercan make the apparatus specify a facial image area more accurately by,for example, selecting the mode that uses all 12 reduction IDs whenphotographing a person who is still.

(20) The specification circuit 184 in the embodiment and themodification was described as specifying a facial image area in acurrent photographed image in accordance with the comparison resultsincluded in two result lists: the result list for the currentphotographed image, and the result list for the photographed image takenone image prior. The specification circuit 184 can be made to specify afacial image area in a current photographed image, however, inaccordance with comparison results included in three or more resultlists. For example, specification can further use the comparison resultsfor a photographed image taken two images prior to the current image. Inthis case, the result memory 181 would also need to store the resultlist for the photographed image taken two images prior.

In the embodiment and the modification, the photographic image inputprior to the current photographic image was described as being thephotographic image taken one image prior, but the current invention isnot limited in this way. A photographic image taken two images priormay, for example, be used. In other words, the apparatus may be modifiedso that it specifies facial image areas included in the currentphotographic image in accordance with comparison results included in tworesult lists: the result list for the current photographed image, andthe result list for the photographed image taken two images prior.

(21) The processor in the embodiment and the modification was describedas selecting, for each photographed image, the reduction IDs forgenerating reduced images from among the 12 reduction IDs. The reductionIDs to choose from, however, are in noway limited to 12 different types.That is, more or less than 12 types may be used.

Also, the processor in the embodiment was described as selecting sixreduction IDs, but of course the number of reduction IDs selected is notlimited to six. As long as the number of reduction IDs selected is equalto or less than the number of reduced images the image processingapparatus is capable of generating within the predetermined timeinterval, the number of selected reduction IDs maybe any number. Thenumber of selected reduction IDs may also be changed for eachphotographed image.

(22) The processor in the modification was described, as shown in FIG.9, as alternatively selecting even numbered reduction IDs and oddnumbered reduction IDs for each photographed image in accordance withthe photographed image counter (steps S2-S4) when no facial image wasdetected in the photographed image taken one image prior to the currentphotographed image (step S20: NO). The processor may be made, however,to choose six or fewer reduction IDs arbitrarily.

(23) The processor in the modification was described, as shown in stepS21 in FIG. 9, as selecting (i) the reduction IDs for which a facialimage was detected in the photographed image taken one image prior and(ii) the reduction IDs in the vicinity of the reduction IDs in (i). Theprocessor may select, however, only the reduction IDs for which a facialimage was detected, or only the reduction IDs in the vicinity thereof.In this case, when the number of reduction IDs is fewer than six, inorder to bring the number of selected reduction IDs up to six, reductionIDs may be selected arbitrarily or selected sequentially from 0 to 11.The same kind of selection can also be made when the number of reductionIDs selected in step S21 in the modification is fewer than six.

(24) In the embodiment and the modification, reduced images for eachphotographed image were described as being generated using six or fewerreduction IDs (referred to in (24) as “selected IDs”) from among 12reduction IDs (referred to in (24) as “candidate IDs”), but themodifications described below are also possible.

(a) When the number of candidate IDs L is an even number and the numberof selected IDs M (M<L) is larger than L/2, then for photographedimages, (i) L/2 even numbered reduction IDs and M−L/2 odd numberedreduction IDs and (ii) L/2 odd numbered reduction IDs and M−L/2 evennumbered reduction IDs can be selected alternately.

For example, if L is 12 and M7, then for photographed images, (i) sixeven numbered reduction IDs and one odd numbered reduction ID and (ii)six odd numbered reduction IDs and one even numbered reduction ID areselected alternately.

Furthermore, the number of even numbered reduction IDs and of oddnumbered reduction IDs selected for photographed images can be adjustedarbitrarily as long as the number of reduction IDs selected for aphotographed image is equal to or less than M. For example, theproportion of even numbered reduction IDs and of odd numbered reductionIDs selected for each photographed image can be changed. In this case,the even numbered reduction IDs and odd numbered reduction IDs that areselected may be selected arbitrarily from among all of the even numberedreduction IDs and odd numbered reduction IDs, or they may be selectedsequentially.

(b) When the number of candidate IDs L is an odd number and the numberof selected IDs M (M<L) is larger than L/2, then for photographedimages, (i) (L+1)/2 even numbered reduction IDs and M−(L+1)/2 oddnumbered reduction IDs and (ii) (L−1)/2 odd numbered reduction IDs andM−(L−1)/2 even numbered reduction IDs can be selected alternately.

In this case, the M−(L+1)/2 odd numbered reduction IDs may be selectedarbitrarily from among all of the odd numbered reduction IDs or may beselected sequentially. The same is also true for the M−(L−1)/2 evennumbered reduction IDs.

For example, if L is 11 and M is 6, then for photographed images, (i)six even numbered reduction IDs and (ii) five odd numbered reduction IDsand one even numbered reduction ID are selected alternately.

Furthermore, the number of even numbered reduction IDs and of oddnumbered reduction IDs selected for photographed images can be adjustedarbitrarily as long as the number of reduction IDs selected for aphotographed image is equal to or less than M as described above in (a).

(c) When the number of selected IDs M (M>L) is smaller than L/2, L beingthe number of candidate IDs, then for photographed images, (i) M orfewer even numbered reduction IDs and (ii) M or fewer odd numberedreduction IDs can be selected alternately. The M or fewer even numberedreduction IDs and the M or fewer odd numbered reduction IDs may beselected arbitrarily from among all of the even numbered reduction IDsand all of the odd numbered reduction IDs respectively, or they may beselected sequentially.

For example, if L is 12 and M is 5, then for photographed images, (i)five or fewer arbitrary even numbered reduction IDs and (ii) five orfewer arbitrary odd numbered reduction IDs are selected alternately.

Furthermore, the number of even numbered reduction IDs and of oddnumbered reduction IDs selected for photographed images can be adjustedarbitrarily as long as the number of reduction IDs selected for aphotographed image is equal to or less than M as described above in (a).

(25) The reduced image generation unit in the image processing apparatusin the present invention corresponds to the resolution conversioncircuit 182 in the embodiment and the modification, the comparison unitcorresponds to the face detection circuit, the specification unitcorresponds to the specification circuit 184, the control unitcorresponds to the processor, and the display unit corresponds to theprocessor, LCD output circuit 150, and LCD 140.

INDUSTRIAL APPLICABILITY

The image processing apparatus in the present invention can be used fordetection of facial images by digital cameras or the like.

[Reference Signs List]

-   100 Image processing apparatus-   101 Processor bus-   102 Memory bus-   110 Camera-   120 Camera input circuit-   130 Image memory-   140 LCD-   150 LCD output circuit-   160 Object detection circuit-   170 Processor-   180 Face detection apparatus-   181 Result memory-   182 Resolution conversion circuit-   183 Face detection circuit-   184 Specification circuit-   200 Semiconductor integrated circuit

1. An image processing apparatus for specifying, in photographed imagestaken at a predetermined time interval and input sequentially, an imagearea as a target of predetermined processing, the image processingapparatus comprising: a reduced image generation unit that (i) hasprocessing capability to generate reduced images, in accordance with aparticular input photographed image, at K (K≧1) ratios within thepredetermined time interval and (ii) is operable to generate reducedimages at ratios indicated from among L (L>K) different ratios; acomparison unit operable to compare each reduced image generated by thereduced image generation unit with template images and to transmitcomparison results; a specification unit operable to performspecification of the image area in accordance with the comparisonresults transmitted by the comparison unit; and a control unit that, foreach photographed image that is input, is operable to (i) select M (M≦K)or fewer ratios for each photographed image in accordance with ratiosindicated for a photographed image input prior to each photographedimage, (ii) indicate the selected M or fewer ratios to the reduced imagegeneration unit and make the reduced image generation unit generate eachreduced image, (iii) make the comparison unit compare each generatedreduced image, and (iv) make the specification unit specify the imagearea for each photographed image in accordance with, at least,comparison results transmitted by the comparison unit.
 2. The imageprocessing apparatus in claim 1, wherein the control unit alternatelyindicates to the reduced image generation unit, as the M or fewer ratiosfor each photographed image that is input, (i) M or fewer ratios thatinclude N (N≦M) ratios selected from among odd numbered ratios and (ii)M or fewer ratios that include N (N≦M) ratios selected from among evennumbered ratios, the even and odd numbered ratios corresponding to the Lratios arranged in ascending or descending order.
 3. The imageprocessing apparatus in claim 2, wherein the control unit (i) selectsone or more ratios from among M or fewer ratios indicated for aphotographed image input prior to a particular photographed image, inaccordance with comparison results transmitted from the comparison unitfor the photographed image input prior to the particular photographedimage, the comparison results indicating that an image corresponding toa template image was detected, and (ii) includes the selected ratios inthe M or fewer ratios for the particular photographed image which areindicated to the reduced image generation unit.
 4. The image processingapparatus in claim 2, wherein the control unit (i) selects one or moreratios from among M or fewer ratios indicated for a photographed imageinput prior to a particular photographed image, in accordance withcomparison results transmitted from the comparison unit for thephotographed, image input prior to the particular photographed image,the comparison results indicating that an image corresponding to atemplate image was detected, and (ii) includes, in the N ratios for theparticular photographed image, ratios that are immediately before orafter the selected ratios in the ascending or descending order of the Lratios.
 5. The image processing apparatus in claim 2, wherein thecomparison unit transmits, as the comparison results, pieces of areainformation each indicating an image area that is detected throughcomparison of each reduced image with template images, the image areacorresponding to one of the template images, and the specification unitperforms the specification in accordance with (i) the pieces of areainformation for a particular photographed image and (ii) the pieces ofarea information for a photographed image input prior to the particularphotographed image, the pieces of area information in (i) and (ii) beingtransmitted by the comparison unit.
 6. The image processing apparatus inclaim 5, wherein the specification unit specifies one image area, inaccordance with pieces of area information that indicate image areasthat are positioned within a predetermined proximity to each other, fromamong (i) the pieces of area information for the particular photographedimage and (ii) the pieces of area information for the photographed imageinput prior to the particular photographed image, the pieces of areainformation in (i) and (ii) being transmitted by the comparison unit. 7.The image processing apparatus in claim 1, wherein the control unitdetermines the M or fewer ratios for a particular photographed image toindicate to the reduced image generation unit in accordance withcomparison results, transmitted by the comparison unit, for aphotographed image input prior to the particular photographed image. 8.The image processing apparatus in claim 1, wherein a plurality ofcandidate template images is stored as candidates for the templateimages, and the comparison unit selects the template images to use forcomparison of a particular photographed image from among the candidatetemplate images, in accordance with comparison results for aphotographed image input prior to the particular photographed image. 9.The image processing apparatus in claim 1, wherein the comparison unittransmits, as the comparison results, pieces of area information eachindicating an image area that is detected through comparison of eachreduced image with template images, the image area corresponding to oneof the template images, and the specification unit performs thespecification in accordance with (i) the pieces of area information fora particular photographed image and (ii) the pieces of area informationfor a photographed image input prior to the particular photographedimage, the pieces of area information in (i) and (ii) being transmittedby the comparison unit.
 10. The image processing apparatus in claim 9,wherein the specification unit specifies one image area, in accordancewith pieces of area information that indicate image areas that arepositioned within a predetermined proximity to each other, from among(i) the pieces of area information for the particular photographed imageand (ii) the pieces of area information for the photographed image inputprior to the particular photographed image, the pieces of areainformation in (i) and (ii) being transmitted by the comparison unit.11. The image processing apparatus in claim 1, further comprising adisplay unit that, for each photographed image that is input, isoperable to display both (i) the photographed image and (ii) a frameindicating an image area at a position on the photographed imagecorresponding to the image area as specified for the photographed imageby the specification unit.
 12. An image processing method in an imageprocessing apparatus for specifying, in photographed images taken at apredetermined time interval and input sequentially, an image area as thetarget of predetermined processing, the image processing methodcomprising the steps of: a reduced image generation step that (i) hasprocessing capability to generate reduced images, in accordance with aparticular input photographed image, at K (K≧1) ratios within thepredetermined time interval and (ii) generates reduced images at ratiosindicated from among L (L>K) different ratios; a comparison step tocompare each reduced image generated in the reduced image generationstep with template images and transmit comparison results; aspecification step to specify the image area in accordance with thecomparison results transmitted in the comparison step; and for eachphotographed image that is input, a control step to (i) select M (M≦K)or fewer ratios for each photographed image in accordance with ratiosindicated for a photographed image input prior to each photographedimage, (ii) indicate the selected M or fewer ratios to the reduced imagegeneration step and make the reduced image generation step generate eachreduced image, (iii) make the comparison step compare each generatedreduced image, and (iv) make the specification step specify the imagearea for each photographed image in accordance with, at least,comparison results transmitted in the comparison step.
 13. An integratedcircuit used in image processing for specifying, in photographed imagestaken at a predetermined time interval and input sequentially, an imagearea as the target of predetermined processing, the integrated circuitcomprising: a reduced image generation unit that (i) has processingcapability to generate reduced images, in accordance with a particularinput photographed image, at K (K≧1) ratios within the predeterminedtime interval and (ii) is operable to generate reduced images at ratiosindicated from among L (L>K) different ratios; a comparison unitoperable to compare each reduced image generated by the reduced imagegeneration unit with template images and to transmit comparison results;a specification unit operable to perform specification of the image areain accordance with the comparison results transmitted by the comparisonunit; and a control unit that, for each photographed image that isinput, is operable to (1) select M (M≦K) or fewer ratios for eachphotographed image in accordance with ratios indicated for aphotographed image input prior to each photographed image, (ii) indicatethe selected M or fewer ratios to the reduced image generation unit andmake the reduced image generation unit generate each reduced image,(iii) make the comparison unit compare each generated reduced image, and(iv) make the specification unit specify the image area for eachphotographed image in accordance with, at least, comparison resultstransmitted by the comparison unit.